{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    ".. _duplication_userguide:\n",
    "\n",
    "Duplicate Values\n",
    "================"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "raw_mimetype": "text/restructuredtext",
    "scrolled": false
   },
   "source": [
    "Introduction\n",
    "------------\n",
    "\n",
    "The function :func:`clean_duplication() <dataprep.clean.clean_duplication.clean_duplication>` creates a user interface that clusters duplicate values and allows the user to merge them into standardized values. The following clustering methods are provided:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### fingerprint\n",
    "This is the process for creating a fingerprint key:\n",
    "\n",
    "* remove leading and trailing whitespace\n",
    "* convert to lowercase\n",
    "* remove punctuation and control characters\n",
    "* normalize extended western characters to ASCII\n",
    "* split into whitespace separated tokens\n",
    "* sort tokens and remove duplicates\n",
    "* join tokens back together\n",
    "\n",
    "### ngram-fingerprint\n",
    "This is the process for creating a n-gram fingerprint key:\n",
    "\n",
    "* convert to lowercase\n",
    "* remove punctuation, whitespace and control characters\n",
    "* get string n-grams\n",
    "* sort n-grams and remove duplicates\n",
    "* join sorted n grams back together\n",
    "* normalize extended western characters to ASCII\n",
    "\n",
    "A textbox is provided for choosing the n-gram size.\n",
    "\n",
    "### phonetic-fingerprint\n",
    "Uses the double metaphone algorithm for generating phonetic-fingerprint keys. The [metaphone](https://github.com/oubiwann/metaphone) library is used.\n",
    "\n",
    "### levenshtein distance\n",
    "Blocking is used to speed up the process, blocks are obtained where strings in the same\n",
    "block share a substring of a given blocking size. Only strings within the same block are\n",
    "compared using the levenshtein distance function. If two values have a distance less than \n",
    "or equal to the given radius they are added to the same cluster. Textboxes are provided for choosing the block size and the radius.\n",
    "\n",
    "The [Levenshtein](https://github.com/polm/levenshtein) library is used for a fast levenshtein distance implementation.\n",
    "\n",
    "Clustering methods are taken from the [OpenRefine](https://github.com/OpenRefine/OpenRefine) project and the [simile-vicino](https://code.google.com/archive/p/simile-vicino/n) project, you can read more about these clustering methods [here](https://github.com/OpenRefine/OpenRefine/wiki/Clustering-In-Depth).\n",
    "\n",
    "The `df_var_name` parameter sets the variable name to be used for the dataframe when creating replace function calls.\n",
    "\n",
    "The `page_size` parameter can be used to set the number of clusters that are displayed on each page of the user interface.\n",
    "\n",
    "Most of the functionality is provided through an interactive user interface which will be introduced shortly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## An example dirty dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "df = pd.DataFrame(\n",
    "    {\n",
    "        \"city\": [\n",
    "            \"Québec\",\n",
    "            \"Quebec\",\n",
    "            \"Vancouver\",\n",
    "            \"Vancouver\",\n",
    "            \"vancouver\",\n",
    "            \" Vancuver \",\n",
    "            \"Toronto\",\n",
    "            \"Toront\",\n",
    "            \"Tronto\",\n",
    "            \"Ottowa\",\n",
    "            \"otowa\"\n",
    "        ]\n",
    "    }\n",
    ")\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "cities = pd.DataFrame(\n",
    "    {\n",
    "        \"city\": [\n",
    "            \"Québec\",\n",
    "            \"Quebec\",\n",
    "            \"Vancouver\",\n",
    "            \"Vancouver\",\n",
    "            \"vancouver\",\n",
    "            \" Vancuver \",\n",
    "            \"Toronto\",\n",
    "            \"Toront\",\n",
    "            \"Tronto\",\n",
    "            \"Ottowa\",\n",
    "            \"otowa\"\n",
    "        ]\n",
    "    }\n",
    ")\n",
    "cities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Default `clean_duplication()`\n",
    "\n",
    "By default the `df_var_name` parameter equals to `default`, which means the prefix of the final result DataFrame is the same with the name of input dataframe. And the `page_size` variable is set to 5. Clustering methods can be toggled using the dropdown menu at the top of the GUI. Select which clusters you would like to merge using the checkboxes under the \"Merge?\" heading. Then press the \"Merge and Re-Cluster\" button to merge the cluster. If the \"export code\" checkbox is selected, code for merging the clusters will be created and added to the notebook cell. Finally, you can press the \"finish\" button to close the GUI and see the final DataFrame created.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dataprep.clean import clean_duplication\n",
    "clean_duplication(df, \"city\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dataprep.clean import clean_duplication\n",
    "clean_duplication(cities, \"city\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. `df_var_name` parameter\n",
    "\n",
    "Pandas Series.replace function calls are created and added to the current notebook cell when merging a cluster with the \"export code\" checkbox selected. This parameter allows for changing the DataFrame variable name used in the exported code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_duplication(df, \"city\", df_var_name=\"dataframe\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. `page_size` parameter\n",
    "This parameter allows for changing the number of clusters that are displayed on each page of the user interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_duplication(df, \"city\", page_size=1)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
